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DCI/RM 81-2313 
12 March 1981 


STAT 
MEMORANDUM FOR: 
irector, Program Assessment Office 


stat moe 


SUBJECT: Report of Ad Hoc Working Group on "Intelligence User Survey" 


1. This memo addresses som technical points about the questionnaire that 
need immediate attention. A future memo will discuss the broader issues of the 
scope of the user survey and the analysis of the resulting data. The 
questionnaire as it stands has some problems that could be resolved with a 
little additional effort. 


The scales and categories used on most questions should be revised to 
insure consistency and elimination of bias and to facilitate analysis 
of the results. 


Questions 2 and 2 are asked in a "yes @ no" format (did you ever 
.».) but the catecories of responses have a “how often" format. 
These questions should be rephrased in a "how often" format. 


Questions 9, 10, 14, and 28 have categories that are not consistent 
with the subcategories in questions 15 through 18. The simplest fix 
would be to change category b of questions 9, 10, 14, and 28 from 
"economic" to "econamic except energy, geographic, @ societal". 


A "not applicable" category should be added to question 9 and 10. 


Question 10 is confusingly phrased as a triple negative. It should 
be rephrased, possibly as "how satisfactory" is intelligence support 
in each topic area rather than "how often" is it unsatisfactory. 


2. The problems with the scales and categories are illustrated by the 
attachment in which the questions are grouped by type of response. It is 
evident that the validity and utility of these categories have not been 
maximized. 


Questions of the "how often" type do not have consistent categories 
foc responses, nar do they have the same number of categories. 


This problem is especially awkward for questions 1, 2, and 3, whose 
responses one would like to campare. Question 1 asks "how often did 
you attempt to review intelligence .. -" and question 2 asks " [how 
often] has intelligence ... directly influenced a policy decision 
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made by you?" It would be quite interesting to see if the user 
answers these questions differently, but without consistent 
categories for responses camparison is difficult. 


In order to avoid eliciting biased responses, whenever possible one 
should use an odd number of categories that are defined symmetrically 
about the middle category. 


Far most questions, five categories are sufficient and more than 
seven could imply distinctions that the user cannot meaningfully 
make. 


The categories "never" and "always" are too narrow for a five point 
scale; 1% of the time is not "never" and 99% of the time is not 
"always". For this reason, some people will avoid using these 
categories far their responses thereby biasing their answers to the 
middle three categories. For ar "how often" questions five 
categories that divide the range roughly equally are preferable, ie: 
0-19% of the events, 20-39%, 40-60%, 61-80%, and 81-100% of the 
events. 


Do “usually", "often", and "mostly" mean the same? Are 
"occasionally" and sometimes" equivalent? Is "often" as close to 
"always" as "rarely" is to "never"? How often is "often"? These 
words are so ambiguous that it is preferable to ancha only the top 
and bottom categories and not use words to describe the intermediate 
categories. 


Questions 1, 2, 3, 9, 10, 20, 24, and 26 ask "how often” in a sense 
meaning fraction of events, but in questions 11, 12, and 19 it means 
number of events per time interval. There is no upper limit to "how 
often" far the latter three questions, while the former eight have 
"always" for a limit. For question 11, "how often did you levy 
requirements ..." does "often" mean daily, weekly, monthly, a@ what? 
For questions 11 and 12 categories similar to those of question 19 
should be used. 


These points apply directly to the question "how satisfied" since it 
has an upper limit, "totally". The questions "how useful", "what 
quality", "how much", and "how important" as used here do not have 
upper limits to their ranges, however, it probably is satisfactory to 
use "extremely useful", "excellent", "excessive", and "extremely 
important" far the top categawies. Again the words fa the middle 
categories should be dropped. 
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how often? (fraction of the events) 


Question 1 


Always 
Usually 
Occasionally 
Rarely 

Never 


Question 2, 9, 10 


Often 
Occasionally 
Rarely 

Never 


Question 20, 26 


Always 

Often 
Occasionally 
Rarely 

Never 


N/A 


Question 3 


Often 
Sometimes 
Rarely 
Never 


how often? (events per time interval) 


Question 19 
Almost daily 


Weekly 
Less often than weekly 


how satisfied? 
Question 25 


Totally satisfied 
Generally satisfied 


Question ll, 12 


Often 
Occasionally 
Rarely 

Never 


Room far substantial improvement 


Completely dissatisfied 


N/A 
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Question 24 


Always 
Mostly 
Sometimes 
Rarely 
Never 
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how useful? 
Question 21 


Extremely useful 
Fairly useful 

Not particularly useful 
Not useful at all 

N/A 


what quality? 
Question 22 
Excellent 
Good 
Fair 
Poor 


N/A 


how much? 

Question 23 
Excessive 
Sufficient 
Insufficient 


N/A 


how important? 
Question 15, 16, 17, 18 


Very important 
Somewhat important 


Not particularly important 


Not at all important 
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Question 27, 28 


Indispensible 

Quite useful 

Not particularly useful 
Not at all useful 


Question 29¢ 


Excellent 
Good 
Satisfactory 
Fair 

Poor. 


Question 29b 


Excessive 

Just about right 

Needed substantially more 
Totally inadequate 
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